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' tn 1.0 Introduction 

oo 

^ This paper describes our research using data from Washington State and Oregon to measure the 

Q benefits derived from the principal labor exchange service offered by the Job Ser\ace: matching job- 

seekers to openings listed by employers. Most matching is done by job-seekers looking through 
computerized lists at Job Service offices. Job-seekers can also view listings using terminals at other 
public places such as libraries and at home using the Internet. 



Sometimes Job Service staff look through listings with individual job-seekers. They also assist 
employers by looking through job-seeker applications to fill specific openings. Those identified then are 
contacted by staff or through automated phone notifications systems and encouraged to apply for those 
jobs. In a few states, including Washington, systematic computerized matches are made weekly between 
job-seeker registrations and all job openings, and then job-seekers are notified that a match has been 
made. 



At one time job-seekers were referred to employers exclusively by the Job Service staff. This 
system still is widely used in Washington. However, many states make it possible for job-seekers to 
obtain contact information through the computer with no staff intervention. Oregon uses this system 
extensively. However, it is the only state that asks job-seekers to identify themselves prior to obtaining 
the contact information. Thus, administrative records in both Washington and Oregon are able to track 
referrals to specific openings. 

For the nation as a whole the Job-Service was credited with referring 7 million different job- 
seekers to jobs in 1999, out of 17 million registrants. Of this group, 2.3 million were placed at the 7.3 
million jobs listed with the Job Service. 

However, these statistics probably substantially underestimate the number of people referred and 
placed by the Job Service because it is has become increasingly difficult for the Job Service to track 
referrals made without staff intervention and the subsequent placements. Evidence that tracking of 
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placements probably declined is that nationwide placements fell by 20 percent, but increased by 2 percent 
in Washington and Oregon, which track placements better than most states. 

Table la displays the proportion of registrants receiving various forms of Job Service assistance 
in 1999. About 60 percent of registrants received some reportable service, with job referral being the 
most common service received. However, a substantial fraction received other forms of job search 
assistance including participation in workshops. Relatively small fractions of registrants received 
assessment services and referrals to services provided by other organizations. 



Table la. Distribution of Services Received by Job Service Registrants 
Reportable service 60.3% 



Referred to job 


37.7% 




Placed 




9.6% 


Job search activity 


33.9% 




Assessment service 


9.2% 




Interviewed 




6.1% 


Counseled 




2.7% 


Tested 




1.4% 


Referred to other services 


7.7% 




Referred to supportive service 




7.7% 


Referred to training or education 




6.1% 



Finally, it is instructive to compare the number of placements made by the Job Service to the number of 
individuals who enter employment by any means after participating in a program funded under the Job 
Training Partnership Act (JTPA). Table lb shows that in 1999 the Job Service placed about 14 times 
more individuals than entered-employment following termination from a JTPA program, at less than one- 
tenth the cost per placement or entered-employment (to the Federal government). The statistics would 
look even more favorable to the Job Service if all PLX users who entered employment, not just those 
placed, were compared to JTPA participants who entered employment. 

On the other hand, JTPA participants often have much poorer prospects for finding work on their 
own than individuals using PLX. Also, the cost statistics omit the substantial state funds that often go to 
the Job Service. Overall, state funds may equal as much as one-third of the Federal Wagner-Peyser share, 
and in Oregon they account for 50 percent of all funding. However, even when the comparisons are 
limited to individuals with similar employment prospects and all costs are taken into account the Job 
Service appears to effectively serve many more individuals at a much lower per-person cost. 
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Table lb. 



Job Service Cost per Placements versus JTPA Cost per Entered-Employment 



Job Service 

Registrants 

Referrals 

Placements 



17,288,127 

7,008,534 

2,304,772 



Wagner-Peyser Allocation $893,000,000 

Cost per person referred $ 1 27 

Cost per person placed $387 



Job Training Partnership Act 

Economically Disadvantaged Dislocated Total 

Adults Youth 
Title II-A Title II-C Title in 



Terminees 202,300 85,000 240,900 

Entered-Employment 151,600 72,000 240,900 



528,200 

464,500 



JTPA Allocation 
Cost per terminee 
Cost per entered- 
employment 



$952,500,000 

$6,284 

$4,708 



$129,600,000 

$1,799 

$1,525 



$1,080,400,000 

$4,485 

$4,485 



$2,162,500,000 

$4,655 

$4,094 



2.0 Measurement Issues 

It is evident that the number of individuals helped by the direct placement services provided by 
the Job Service are substantially greater than the number helped through other Job Service aid and dwarfs 
the number helped through JTPA programs. What is not at all evident is how the per-person benefits vary 
across the "treatments" shown in table la. 



There are several studies that provide accurate estimates of the value of job search assistance for 
dislocated workers, unemployment insurance (UI) claimants, and the economically disadvantaged, as well 
as the benefits of classroom vocational training and on-the-job training for these groups. Importantly, 
none of the studies explicitly examine the role of Job Service job matching, although much of the benefits 
of the various treatments probably come from participants' use of those services. Even studies of job 
search assistance focus on workshops usually lasting only a few days, and do not get inside the "black- 
box" to determine whether the source of the benefits is increased use of Job Service referrals or more 
effective finding of jobs through other means. 
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Despite their shortcomings with respect to the role of job-matching, these studies provide 
persuasive evidence about the net benefits of the treatments examined because they are based on 
experimental evidence derived from random-assignment designs. In broad outline, these studies require 
program operators to use normal channels to enlist more volunteers interested and qualified for 
participation than the programs can accommodate. They then randomly select a control group of 
individuals who are not permitted to participate in these programs for a set period. 

Estimates of these programs’ "value-added" are then obtained through straightforward 
comparisons of the post-program outcomes of the participants to what subsequently happened to members 
of the control group. Use of the random-assignment design ensures that whatever factors affect the 
decision to volunteer for a given treatment and also affect post-program outcomes are distributed identical 
between the targets (participants) and controls. 

It has been well-documented that individuals volunteering to participate in government funded 
employment and training programs are drawn to the programs because they have unusual difficulty 
achieving their career goals. However, once factors associated with such difficulty are taken into account 
it often turns out that the volunteers differ from apparently similar non-volunteers because they have 
greater motivation to succeed, and therefore, would have had better outcomes even in the absence of 
participation. 

Table 2 presents an example of how both negative and positive self-selection bias makes it 
difficult to draw accurate conclusions from simple comparisons among UI claimants who registered with 
the Washington State Job Service between 1987 and 1995 and changed jobs (rather than were recalled to 
former jobs or dropped out of the labor force). 



Table 2. Probability of Claimants who Change Jobs Returning to Work 



Referred 



Return after: 


Not Referred 


Not Placed 


Placed 


1 quarter 


49.0 


41.7 


63.5 


2 quarters 


67.3 


66.8 


86.2 


3 quarters 


78.2 


86.2 


92.4 



As might be expected, in all three periods those placed at jobs were more likely to return to work 
than those referred-but-not-placed and those not-referred. However, in the first quarter following the start 
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of a period of joblessness, not-referred claimants were about 25 percent more likely to return to work than 
those who foimd jobs to which they were referred, but did not end-up being placed at those jobs. 

This evidence suggests that within the first 13 weeks of unemployment individuals who did not 
use the Job Service were better able to find jobs on their own than those who did. Thus, there was 
negative selection on the part of Job Service users. However, the same comparison in quarter 3 suggests 
that Job Service users were about 10 percent more likely to return to work. This 10 percent increase 
could reflect better qualifications and more interest in finding work, or the benefits of using the Job 
Service. 



Further, if we wanted to know how likely job seekers placed in the first quarter were to find jobs 
in the absence of use of the Job Service, would it be better to compare those placed to those referred or to 
those not referred? Without some type of randomization among those "treated" versus those not treated it 
is extremely difficult to determine what would have happened in the absent of a given treatment. 

The primary challenge in conducting an analysis of the value-added of direct placement services 
is that, by law, the Job Service must help all comers, and by its Very nature, it would be extremely 
difficult to prevent job-seekers from viewing listings. We, therefore, cannot use the random assignment 
technique that has been successfully employed in measuring the effectiveness of classroom training and 
job-search workshops, to assess the Job Service "treatment" of viewing listings. 

What we were able to do, however, was employ a promising approach that takes advantage of a 
natural source of variation to measure the value of being placed relatiye to being referred-but-not-placed. 
This "quasi-experiential" approach underestimates the total benefits of using the Job Service because it 
omits the value of obtaining information from viewing the listings and pursuing promising openings that 
ends up helping these users quickly find and accept other jobs. However, once we have an accurate 
measure based on a quasi-experimental design, we can then use that measure as a benchmark for 
developing a non-experimental estimation procedure that produces accurate results. 

3.0 Use of a Quasi-Experimental Design 

The key to developing an imbiased measure of the value of placements relative to referrals-not- 
leading-to-placements is finding a naturally occurring source of random variation in the probability of 
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being placed following obtaining a referral to a particular opening. We felt that such natural variation 
was likely to occur because there was a lag between when an employer made a job offer to an opening 
listed with the Job Service and when that listing was removed from the Job Service computers.^ As a 
result, some referred job-seekers applied to openings after they were filled, and therefore through no fault 
of their own, could end-up without job offers. 

Applying to a job that is already filled is similar to the following experiment (that could not 
actually be conducted). Employers who list jobs with the Job Service would be required to request the 
Social Security Account Number (SSAN) of each applicant and ask how they heard about the job. If the 
job-seeker's SSAN ends with 2 and the person heard about the job from the Job Service the employer 
would be required to tell the job-seeker the job has already been filled.^ 



We set out to identify referred job-seekers who applied too late to obtain job offers by mailing 
surveys to 6,000 job-seekers in Washington State, half of whom were placed, and half referred to the 
same jobs, but not placed.^ We used a variety of incentives to secure a maximal response, but only 
obtained a 22% response rate. Thus, our sample may not be representative of the universe, and created 
larger confidence intervals for the point estimates than we would like. Nevertheless, the sample was 
sufficient to test the viability of the approach. 



Table 3 presents the main results of the survey. Column 1 describes the outcomes following 
obtaining specific referrals. To our knowledge this is the only information available on this topic. 



* Lags exist because employers do not immediately notify the Job Service when a job-offer is made. Our own 
survey of job-seekers, interviews with Job Service staff, and interviews with employers suggested that it is not 
simply that employers are too busy to notify the Job Service, but that job-seekers often take some time to decide to 
accept on offer, do not show up for work after accepting an offer, and quickly stop working after they discover the 
job is not to their liking or more difficult than they anticipated. Thus, employers often find it useful to keep adding 
to their list of potential hires, even after an offer has been made. Also, many jobs are filled quickly (within a week 
or two of being listed). Thus, even short lags in their removal can generate referrals to filled jobs. 

^ As already noted, a better experiment would be to randomly deny access to the list of Job Service openings 
because then our measure would include benefits from viewing the lists and either not finding suitable opening or 
not obtaining a job from a suitable opening. Such an experiment is impossible, and because those benefits are 
omitted, our quasi -experimental design produces conservative estimates. 

^ A key difference between the natural experiment we used and the experimental design sketched above is that when 
a job-seeker is not granted an interview to one job (because it is already filled) he or she can gain interviews as a 
result of other referrals. To deal with this issue we compared placed individuals who were referred to the same job, 
but not placed at any job. We estimate that removing those subsequently placed only reduced the pool of those 
referred-but-not-placed by one-eighth. 
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28. 1 percent did not attempt to secure an interview. 34.4 percent tried to obtain an interview, but were 
unable to do so. 38.5 percent secured an interview, but of this group, only 40.0 percent were placed. 32.6 
percent were not given an offer, and 10.2 percent either rejected an offer, or accepted the offer and later 
decided not to report for work. 



Table 3. The Distribution and Effect of Referral Outcomes. 



Outcome 


Percent of 
Job-Seekers 


Change in 
unemployment 


1 . Placed (started work) 


15.0 


duration (weeks) 
-4.70 


2. Accepted offer, no show 


4.6 


-1.25 


3. Rejected an offer 


5.6 


-6.52 


4. Interviewed, did not receive offer 


12.2 


1.65 


5. Tried, but failed, to get an interview 


34.4 


— 


6. Did not follow-up referral 


28.1 


-1.36 


Groups 1 through 4 relative to 5 


38.5 


-2.50 



We feel that a comparison between those who obtained interviews and those who tried but failed 
to obtain an interview provide evidence that is reasonably close to that which would be derived from a 
random-assignment design. We hold this view because we believe that most of these job-seekers who 
tried, but failed to secure an interview, applied for the jobs too late to get an interview, and therefore, 
comprise a quasi-control group. 



Unfortunately, the purity of the quasi-control group is not certain. Some individuals could have 
failed to secure interviews because they were screened out by employers, and therefore, should not be 
included in the comparison group. In retrospect, we should have probed deeper to determine whether pre- 
screening occurred, and have subsequently designed a survey instrument that should be able to limit the 
comparison group to job-seekers who applied too late to obtain interviews. 

Column 2 of table 3 shows the difference in weeks between the duration of joblessness of 
individuals in each category and those who tried but failed to obtain an interview. The results show that 
those placed returned to work 4.70 weeks sooner than those who tried but failed to obtain an interview. 
Those who rejected an offer returned to work 6.52 weeks sooner. Presumably, these rejections were due 
to having a better offer in hand or lined up. Also, much in keeping with expectations, individuals who 
failed to get an offer, took 1.65 weeks longer to find work. 
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On average, job-seekers who interviewed returned to work 2.50 weeks sooner than those who 
tried, but failed to obtain an interview. It is this finding that resembles the results of a true experiment. 

In addition, to computing the effect for all job-seekers together we separately examined what 
happened to job-seekers who had "spotty" work records (employment in 0, 1, or 2 quarters in the year 
prior to the start of their unemployment spell), and job-seekers who had "strong" work records 
(employment in 3 or 4 quarters in the year prior to their unemployment spell). 

The distribution of outcomes is very similar to those shown in column 1 of table 3 for both 
groups. Thus, are omitted from table 4. The effect of being placed (relative to being unable to secure an 
interview) also was similar for both groups. However, the difference in weeks of unemployment across 
the other outcomes is very different in all other cases. 

Table 4. The Effect of Referral Outcomes for Job-Seekers with Spotty and Strong Work-Records. 





Spotty 


Strong 


1 . Placed (started work) 


-4.10 


-4.22 


2. Accepted offer, no show 


-5.64 


-2.66 


3. Rejected an offer 


-2.97 


-7.35 


4. Interviewed, did not receive offer 


4.29 


0.44 


5. Tried, but failed, to get an interview 


— 


— 


6. Did not follow-up referral 


-3.02 


2.62 


Groups 1 through 4 relative to 5 


-1.17 


-2.77 



On average, job-seekers with spotty records who interviewed for a referred job returned to work 
1.17 weeks sooner than job-seekers who tried but failed to obtain an interview. Job-seekers with strong 
work records who interviewed returned to work 2.77 weeks sooner than job-seekers unable to secure 
interviews. 

The substantially greater effect of being interviewed for those with strong versus spotty work 
records mainly stems from the those with strong records who failed to obtain an offer returning to work 
only slightly later than those unable to secure an interview. In contrast, those with spotty work records 
who failed to obtain an interview returned to work over 4 weeks later than those unable to obtain an 
interview. 
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The large difference in unemployment duration between those with spotty work records who did 
not obtain offers versus those unable to obtain interviews suggests that being screened out by one 
employer makes it likely that other employers also would not hire the job-seekers with spotty work 
records. This result suggests that including individuals in our quasi-control group who were screened out 
by employers prior to being offered a chance to interview could lead to substantial overestimation of the 
effect of being placed for this group. 

In sharp contrast, the small difference between those with strong work records who did not 
receive offers and those unable to secure an interview suggests that being screened out by employers does 
not have much of a negative effect on the duration of job search. This finding suggests that even if some 
of those who were unable to secure interviews were screened out by employers it would lead to a small 
overestimation of the true effect for individuals with strong work records. This is an important finding 
because it suggests that the measures developed from our survey for job-seekers would be relatively 
unaffected even if there were substantial numbers of job-seekers screened out by employers among those 
unable to secure an interview. 

It also is worth noting that our initial expectation was that the effect of referrals would be smaller 
for workers with strong work-records than those with spotty work-records. Our reasoning was that 
workers with strong-records would have better access to information sources, such as leads from fhends 
and relatives, that would help these individuals more quickly finding suitable jobs without Job Service 
assistance. However, it appears that this hypothesized lack of information sources is more than offset by 
job-seekers with spotty work-records being willing to accept a far greater range of jobs, most of which 
require little training or experience.'* 

3.1 Use of Quasi-Experimental Estimates as Benchmarks 

The final exercise we conducted using the survey data was to compare the results based on a 
quasi-experimental design to those that would be generated using non-experimental estimators. The non- 



If we are correct that job-seekers with strong work-records are looking at a narrow range of jobs, than it suggests 
that it is important to compare placed job-seekers with strong work records to similar job-seekers who are referred 
but not placed at those same jobs. Also, it suggests that for job-seekers with strong work records it is relatively 
unlikely that there will be a lot of suitable openings. Thus, our inability to prevent job-seekers applying too late to 
be placed at one job from gaining interviews from other referrals is unlikely to have a large effect on the estimates. 
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experimental estimator we used compares the difference in the duration of joblessness between job- 
seekers who are placed (those in group 1 in tables 3 and 4) versus all those who are referred but not 
placed (those in groups 2 through 6), 

In the absence of detailed survey information about what happened following the receipt of a 
referral this is the only meaningful comparison that can be made, A regression is used to hold constant 
the observed differences between those placed and those referred but not placed. 

To create a comparable quasi-experimental estimate we assume that the effect of the referral on 
those not placed is zero, and that the only effect is on those placed. Doing this simply requires weighting 
the estimates displayed in tables 3 and 4 by the proportion of interviewed individuals who are placed. 
Fortunately, precisely the same assumption about the value of being referred-but-not-placed equaling zero 
underlies those estimates. As a result, both estimates are comparable, and both do not include any 
benefits stemming from viewing listings, but not being placed. 

Line 1 of table 5 shows the quasi-experimental estimates and line 2 shows the proportion of job- 
seekers who are placed following an interview. Dividing the estimates on line 1 by the proportions on 
line 2 produce the quasi-experimental estimates shown on line 3 (based on the assumption that there is no 
value-added from viewing listings unless they lead to a placement). 



Table 5. Comparisons between Quasi-Experimental and Non-Experimental Estimates 



Quasi-Experimental Result 
1 . Interviewing and possible placement versus 
trying, but failing to secure an interview 


Work-Record: 
Spotty Strong Both 

-1.082 -2.761 -2.383 


2. Fraction of those interviewing who are placed 


.380 


.368 .374 


3. Placement effect (line 1 / line 2) 


-2.849 


-7.510 -5.978 


Non-Experimental Result 
4. Placement versus referral but no placement 


-3.411 


-4.496 -4.046 


Difference quasi-experimental versus non-experimental 

5. Weeks 

6, Percent of non-experimental result 


.563 

-16:5% 


-3.013 -1.933 
67.0% 47.8% 
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Line 4 of table 5 shows the non-experimental estimate based on comparing those placed to those 
referred-but-not-placed. Line 5 shows the difference between the two estimators in weeks of 
unemployment and line 6 shows the difference as a percent of the quasi-experimental estimator. 

The key conclusions from this table is that for job-seekers with strong work records the non- 
experimental estimates are about 67 percent less than the quasi-experimental estimates, and for Job- 
seekers with spotty work records the non-experimental estimates are about 17 percent greater than the 
quasi-experimental estimates. 

Our view is that this comparison provides a useful assessment of the bias in the non-experimental 
estimates for job-seekers with strong work-records. We hold this view because we doubt that including 
some individuals in the quasi-control group who were screened-out by employers had much of an effect 
on the quasi-experimental results. 

In contrast, we feel that is far more difficult to use the quasi-experiential results to assess the bias 
in the non-experimental estimator for job-seekers with spotty work-records. As previously discussed, the 
large effect of not getting an interview for this group makes it far more likely that the quasi-experimental 
results themselves over-estimate the true effect. 

Nevertheless, we are inclined to accept both the much smaller effect of placements on job-seekers 
with spotty work records and that the non-experimental estimators tend to over-estimate the true effects. 
Both these surmises stem from the estimate that job-seekers with spotty work-records are unemployed 
3.02 weeks less than those who fail to get interviews and only 1 .08 weeks longer than those who are 
placed. 



More specifically, it is our view that the small difference between being placed versus not 
following-up the interview arises because job-seekers with spotty work-records (who use the Job Service) 
have access to other sources of leads that are almost as good as those available from the Job Service, and 
therefore, the value-added of the Job Service placements should be relative small. 



3.2 Summary of the Survey Results 

An expert panel reviewed our survey results and agreed on four main conclusions: 
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1 . The survey results are imperfect because we did not rule out the possibility that some job-seekers 
who tried but failed to secure interviews were screened out by employers. 

2. The sample may not have been representative of the universe of placements and was small 
leading to large confidence intervals surrounding some point estimates. 

3. These problems should be able to be overcome by improving the survey instrument to determine 
if employers pre-screened job applicants and by using a telephone follow-up to obtain a larger 
and more representative sample. 

4. Thus, in contrast to earlier efforts, our quasi-experimental design holds substantial promise for 
obtaining credible estimates of the value of being placed that also can be used as benchmarks in 
assessing the bias in non-experimental estimators.^ 

Also, despite the above shortcomings, the survey directly provided useful information about: 

o Why only about 15 percent of job-seekers referred to an opening are placed at that job. 

o How long it takes job-seekers with different referral outcomes to return to work. 

o How the outcomes and the duration of joblessness differ between job-seekers with strong work- 
records (3 or 4 quarters with employment prior to becoming unemployed) versus spotty work- 
records (less than 3 quarters with employment). 

This information, in turn, made it possible to compare two alternative ways to estimate the effect 
of placements stemming from Job Service referrals on the duration of joblessness, and assess the accuracy 
of the two procedures: 



o The quasi-experimental design compared job-seekers who tried, but failed to secure interviews to 

those who succeeded in gaining interviews. On average, this design produced estimates that 
interviewing for a referred job reduced joblessness by 2.76 and 1.08 weeks for job-seekers with 
strong and spotty work records, respectively. If we assume that only those placed benefited from 
gaining interviews, on average this design produced estimates that placements reduced 
joblessness by 7.51 and 2.37 weeks for job-seekers with strong and spotty work records, 
respectively. 



^ The expert panelists also were skeptical that the effects of placements and referrals were so much larger than 
estimates of the effects of job search assistance (ISA) generated from random-assignment designs. However, it is 
our view that this criticism stemmed from key differences between the treatment offered by the Job Service and the 
JSA treatments analyzed using random-assignment designs. Those studies tended to examine the effect of 
participation in a workshop that usually lasted well less than 5 days. 

We suspect that the positive effects were narrowly distributed among a small segment of participants who 
used the information they obtained to change the way they would otherwise have searched for work. However, the 
design averaged the effect across all participants. In contrast, our approach assumed the entire effect was on the 15 
percent who were placed. 

In addition, as will be discussed in the next section, the period over which job-seekers actively used the Job 
Service to look at openings was much longer than the duration of the workshops. Thus, we believe the treatment 
provided by the Job Service should be expected to produce much larger results than a one- week workshop that was 
not followed-up with additional actions by many participants. 
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o The non-experimental design compared job-seekers who were placed to those who were referred, 
but not placed. On average, this design produced estimates that placements reduced joblessness 
by 4.50 weeks and 3.42 weeks for job-seekers with strong and spotty work records, respectively. 

o It is our view that the quasi-experimental estimate is relatively free of bias for job-seekers with 
strong work-records. Because individuals who interviewed and did not receive job offers are 
unemployed only .44 weeks longer than members of the quasi-control group there appears to be 
little correlation between not obtaining an offer and subsequent unusual difficulty finding work. 
This makes us believe that even if employers are screening out substantial numbers of the quasi- 
control group (who tried but failed to obtain interviews), it would not have much effect on that 
groups jobless duration. 

o Based on the same reasoning we have more doubt that the quasi-experimental estimates for job- 
seekers with spotty work-records are relatively free of bias. For this group, job-seekers who 
failed to get offers were unemployed 4.29 weeks longer than members of the quasi-control group. 
Thus, if those being screened-out prior to being interviewed had similar durations to those 
screened-out after being interviewed, large over-estimates would occur if employers screened out 
many job-seekers prior to being interviewed. 

o Because we feel that the quasi-experimental results are relatively free of bias, we believe that the 
non-experimental estimator substantially under-estimates the true effect. However, we would 
prefer to have had a larger and more representative sample on which to base this assessment. 

Overall, the authors of this report and the expert panelists agree that obtaining unbiased estimates 
from a quasi-experiment design is crucial to accepting the validity of results derived from any non- 
experimental design. We also agree that the quasi-experimental evidence we generated for job-seekers 
with spotty work records is of limited use for assessing the value of placements or judging the validity of 
non-experimental estimators. 



We disagree, however, on how much confidence can be attached to our quasi-experimental point 
estimates for job-seekers with strong work records and our conclusion that the non-experimental 
estimates under-estimate the true effect for that group. 

Basically, the panelists felt that the main contribution of our work was to demonstrate that it was 
highly worthwhile to improve the survey to the point it produced highly credible estimates. Their view 
was that only improved results should serve as benchmarks forjudging the direction (and magnitude) of 
the bias in non-experimental estimates. 
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We regard having the expert panelists agree that it is possible to produce credible estimates using 
our quasi-experimental design as a considerable achievement. However, we also feel that the existing 
evidence is sufficient to believe that the non-experimental estimates of per-person placement effects for 
claimants represent conservative estimates of those effects. We, therefore, feel that is worthwhile to 
present our results for this group in the next section.^ 

We even more strongly hold the view that assuming referrals that do not lead to placements have 
no value leads to a highly conservative estimate of the total value of referrals. Thus, we feel that it 
worthwhile to present the benefit-cost estimates in section 5. 

4.0 Non-Experimental Estimates Using Administrative Data 

This section describes our analysis of the effects of referrals and placements made to 
unemployment insurance (UI) claimants in Washington and Oregon. These estimates used non- 
experimental design with exceptionally large administrative databases. 

We restricted the analysis to claimants because: 

1 . Claimants must attest that they did not quit their former jobs and that they are actively 
searching for work. 

Also, the database is unusually rich for claimants. Thus, we can determine: 

2. What labor exchange services were received and how many weeks of joblessness preceded 
their receipt. 

3. How many quarters of unemployment followed obtaining a given referral. 

4. Whether a claimant took a new job, rather than ended a spell of unemplo)mient by returning 
to work with the employer that laid him or her off, or dropped out of the labor force. 

Thus, these databases allow us to compare placed claimants to other claimants who are looking 
for new jobs, and are similar in other key respects, including whether and when they obtain referrals. 



^ The experts differed in the emphasis they placed in the usefulness of having these estimates versus estimates that 
could be much improved. Thus, taken together, there is a paradoxical flavor to the reports in that they suggest both 
that the estimates we produced are the best available and that they are not good enough to be taken as reasonably 
close to the true values. 
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Conceptually, we expect that selection bias will be greatly reduced by controlling for choosing to 
use labor exchange services, finding openings to which the person wants to be referred, and being jobless 
for a comparable period at the time of referral. Together with controlling for age, sex, prior earnings 
level, tenure, occupation, industry, and local labor market conditions we find it quite plausible that our 
comparisons are being made among job-seekers with similar reemployment prospects. 

In addition, most claimants must register with the Job Service, but are not required to use the Job- 
Service to search for work. Thus, we have information about a comparison group of claimants who did 
not obtain referrals that can be used to measure the benefits of obtaining referrals that do not lead to 
placements. Self-selection bias is more likely to be present in comparisons between those referred and 
those who chose not to look at listings, than between those placed and referred. However, we think that 
the control variables we have available to us should substantially reduce such bias. In particular, we 
believe that comparing claimants who are referred to claimants who are not referred but have similar 
durations of joblessness should go a long way to reducing selection and other sources of bias. 

4.1 Measures of Placement and Referral Effects 

Table 6 describes our estimates of the effect of being placed and being referred in each of five 
periods relative to the start of claimants' spells of joblessness. As noted in the preceding subsection, we 
feel that confining the comparisons to claimants who were unemployed for the same duration at the point 
services were received should substantial reduce selection bias. We also controlled for a host of other 
factors. (Appendix A describes the variables included in the estimating equations, and displays the results 
of using our model) 

In Washington, we estimate that, on average, each referral not leading to a placement reduced 
joblessness by 2.149 weeks and each placement reduced joblessness by 9.867 weeks.’ Referral effects 
rose slightly peaking in weeks 10-13, and then declined. Placement effects rose sharply over the first 
three periods, and also peaked in weeks 10-13, but only declined slightly thereafter. 



’ We measured joblessness using quarterly wage records because once claimants exhausted UI benefits we could no 
longer use UI payment measures alone, and many claimants would have exhausted benefits in the absence of being 
referred. Rather than use a hybrid measure of weeks prior to exhaustion and quarters times 13 after exhaustion we 
determined that multiplying quarters times 13 and adding 4 closely correlated with weeks of unemployment during 
the pre-exhaustion period. 
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Table 6 Estimates of Referral and Placement Effects 



Referral/Placements 
made in: 



— Effect on weeks of joblessness — 
referrals placements placements 

(relative to no referral) (relative to 

referral no 
placement) 



A. Washington 1987-95 



week 1 


-2.029 


-5.569 


-3.540 


weeks 2-9 


-2.340 


-8.427 


-6.087 


weeks 10-13 


-2.460 


-12.388 


-9.928 


weeks 14-18 


. -2.175 


-11.867 


-9.692 


weeks 19-26 


-1.399 


-12.187 


-10.788 


average of 


-2.149 


-9.867 


-7.718 


all weeks 








Oregon 1995 








week 1 


-2.292 


-5.499 


-3.207 


weeks 2-9 


-0.171 


-5.223 


-5.052 


weeks 10-13 


-2.781 


-6.943 


-4.162 


weeks 14-18 


-2.550 


-5.981 


-3.431 


weeks 19-26 


-1.293 


-9.236 


-7.943 


average of 


-1.131 


-5.751 


-4.620 



all weeks 



C. Incidence of Referral and Number of Claimants in our Sample 

Percent of Claimants Referred % Claimants Remaining 

Oregon Washington Oregon Washington 



week 1 


2.9% 


2.3% 


100.0% 


100.0% 


weeks 2-9 


12.0% 


4.6% 


83.6% 


89.0% 


weeks 10-13 


4.6% 


2.8% 


42.9% 


51.4% 


weeks 14-18 


4.2% 


3.4% 


32.9% 


40.6% 


weeks 19-26 


5.5% 


5.0% 


24.0% 


29.8% 


Average of all weeks 


5.9% 


3.4% 


Number in sample 

138,280 


328,815 



Note: The Washington sample includes 20 percent of all claimants who changed Jobs. The Oregon sample includes 
100 percent of those claimants. Claimants who return to a former employer or do not show any employment within 
7 quarters of the start of unemployment spells are not included in either sample. Also, we excluded referrals and 
placements that occurred after the claimants stopped collecting benefits or reached their 26th week of 
unemployment (when most claimants exhaust their benefits). Thus, our samples only include ’’active” claimants. 
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The placement trend indicates that claimants are able to far more readily find jobs by other means 
close to the start of their jobless period, but as they exhaust the most promising leads, it becomes 
increasingly more difficult to find jobs without Job Service referrals. Similarly, it is our view that the 
decline in the value of referrals not leading to placements stems from job-seekers first use occurring in an 
earlier period and the value of the information gained by viewing listings and contacting employers but 
not being placed declining with subsequent use. Thus, placements become increasingly more important 
as jobless duration lengthens. 

The increase in the effect of placements relative to referrals is shown in column 3 of table 6, 
which displays the difference between the placement and referral effects. We also regard these figures as 
conservative estimates of the placement effects based on assuming that the entire referral effect is 
capturing selection and other measurement biases. 

For Oregon, we estimate that, on average, each referral not leading to a placement reduced 
joblessness by 1.131 weeks and each placement reduced joblessness by 5.751 weeks. In both cases the 
effects are substantially less than those measured for Washington. However, the pattern of referral effects 
are similar to those in Washington except during weeks 2-9. The pattern of placement effects are not 
especially similar accept for week 1 and weeks 19-26. 

We examined a number of explanations for why the effects are different in the two states. We 
believe that some of the difference is due to the Washington data covering a period where jobs were 
difficult to find and extended benefit programs were in place. As a result, in the absence of placement, 
unemployment would be much longer. However, we also believe that differences in requirements placed 
on Washington and Oregon claimants and funding to ensure the requirements were met were of great 
importance. More specifically, Oregon used its own funds to call-in many more claimants to be 
interviewed by Job Service staff to ensure they are complying with the UI work-test. We believe that 
once an office visit is made it is natural for claimants to look at job orders and obtain referrals. 

Panel C of table 6 confirms that a much larger fraction of claimants are referred in Oregon than in 
Washington. The difference in favor of Oregon is especially large during weeks 2-9, when mandatory 
registration is required and screening is most common. In contrast, the difference is smallest in weeks 19- 
26 when it is most likely claimants in both states are using the Job Service because other methods to 
locate jobs have failed. Thus, we feel that is not at all coincidental that the biggest differences between 
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the two states occurred in weeks 2-9, when usage differed the most, and the smallest differences occurred 
in weeks 19-26, when usage differed the least. 

If our hypothesis is correct that Oregon’s UI collection requirements led many more claimants to 
begin using labor exchange services early during their unemployment spell and feel more pressure to 
assiduously search for work or loose UI benefits; then it is reasonable to expect that Washington 
claimants who take the trouble to use the Job Service are more in need of help, and therefore, likely to 
show greater benefits per-person. Similarly, it is reasonable to expect that Oregon claimants who do not 
obtain referrals search harder by other means, and thus, reduce the measured difference between users and 
non-users of labor exchange services. 

Our data do not prove that UI rules lead to claimants returning to work or ending their spell more 
quickly in Oregon than Washington. However, it is clear that claimants in Oregon return to work more 
quickly. As shown in columns 3 and 4 of panel C, over 5 percent fewer Oregon claimants who were 
unemployed for 1 week continue into the 2nd week. Also, only 42.9 percent of Oregon claimants 
continue into the 1 0th week, compared to 5 1 .4 percent of Washington claimants. 

Finally, what ever the reason for the quicker return to work and earlier use of the Job Service in 
Oregon, this pattern makes it is more difficult to control for the effect of unobserved characteristics. This 
is the case because elapsed duration is one of the most powerful predictors of "what otherwise would 
happen". Thus, we believe that the affect of selection bias leads to greater underestimation of the true 
effect in Oregon than in Washington. 

4.2 Accuracy of the Estimates 

The fundamental question concerning the estimates displayed in Table 6 is whether they are 
sufficiently close to the true values for the evidence to be useful for policy making purposes. We believe 
that our results are reasonably close to the true value for a variety of reasons. In particular, we feel that 
the estimates of the reduction in the period of joblessness due to being placed versus being referred-but- 
not-placed provides a conservative estimate of the true effect. 

Our best evidence for this view is that the quasi-experimental estimates of being placed relative to 
being referred-but-not-placed for workers most like claimants are remarkably close to the non- 
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experimental estimates for claimants. The quasi-experimental estimate for job-seekers with strong work 
records is -7.510 weeks, compared to the non-experimental estimate for claimants of -7.718 weeks. 
Moreover, use of a non-experimental estimator similar to the one used in this section produced estimates 
that were even smaller than the quasi-experimental estimators making us believe that such estimators 
produce conservative estimates. 

We have already noted that the best way to judge the validity of the non-experimental estimates is 
to compare them to the results of a properly executed random-assignment design. We also made clear 
that we concur with the view expressed by the expert panel that the quasi-experimental evidence is not 
conclusive and executing a new survey would be extremely useful. 

However, we also feel that it would be a mistake to take the extreme view that since we cannot be 
certain the quasi-experimental estimates are unbiased we should assume that they are highly biased. After 
all, the internal evidence in the survey suggests that our estimates for job-seekers with strong work- 
records would not be seriously biased even if our comparison group included job-seekers screened out by 
employers.® 

The bottom-line is that we need additional quasi-experimental evidence to effectively refute 
claims that our estimates might enormously overstate the true value of a placement relative to a referral 
not leading to a placement. Nevertheless, we feel that it is highly plausible that our estimates are good 
first-approximations of the true effects, or put another way, it is plausible that job-matching services 
leading to placements have strong positive effects. 



® The experts did not claim that they had evidence either that being screened out increased jobless duration or that 
many screened out job-seekers were in the quasi-control group used to estimate placement effects for job-seekers 
with strong work-records. What they claimed is that both hypotheses were sufficiently plausible that they can and 
should be ruled out . 

We believe that it is plausible that some job-seekers with poor work records lacked good qualifications and 
were screened out by employers based on characteristics that are not observable directly or indirectly in our data, 
and that those job-seekers would have more difficulty finding work than job-seekers with similar observable 
characteristics. However, we also think it is plausible that an equal number are screened out because they are over- 
qualified and would not accept a job even if one was offered (and presumably would have less difficulty finding 
work than job-seekers with similar observable characteristics). 

Importantly, this counter-argument only reinforces the basic point that having evidence that experts agree 
comes from a properly executed experimental or quasi-experimental design is invaluable because it tends to 
eliminate speculation about the myriad of possible ways results could be biased in one way or another. 
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First, it is clear that placements lead to a quick return to work. The question is how quick would 
have the return to work have been otherwise. Similarly, it is evident that at any point during a spell of 
joblessness relatively few claimants return to work within the subsequent five weeks.^[9] Thus, over- 
estimates can only come about by the claimants who are placed being in the small group of claimants who 
would have rapidly found jobs by other means. 

While we cannot prove that the non-experimental estimators eliminated much of the selection 
bias that was present, the comparisons presented in table 7 show that the estimators substantially modified 
the estimates in the appropriate direction. 



Table 7. Mean Differences and Regression Adjusted Differences in Joblessness 





Referred-not-placed 
Versus not-referred 


Placed versus 
Not-referred 


Placed versus 
Referred-not-placed 


Mean difference in 
unemployment duration 


.208 


-12.254 


-12.462 


Regression adjusted 
difference 


-2.149 


-9.867 


-7.718 


Effect of regression 
adjustment 


2.357 


-2.387 


-4.744 



Column 1 shows that the simple difference in mean duration between those referred-but-not- 
placed and those not-referred is .208 weeks, but when the regression model is applied the difference 
becomes -2.149 weeks. This implies that those not referred who most closely resemble those who were 
referred-but-not-placed were unemployed about 2.357 weeks longer on average. This difference is 
consistent with there being negative selection associated with Job Service use. 



Column 2 shows that the regression model reduces the mean difference in duration between those 
placed and those not-referred from -12.254 weeks to -9.867 weeks. This implies that those not placed 



^ Claimants can afford to be quite selective in the jobs they interview for and accept, especially in the first 5 to 10 
weeks of their unemployment spell. Indeed, a key purpose of UI benefits to allow recipients the time they need to 
locate the most suitable job among those available. Thus, the presence of UI payments will lead claimants to remain 
unemployed for substantial periods, unless they find highly suitable jobs. As claimants get closer to exhaustion, 
however, they are more likely to settle for jobs that they would not take earlier, but their inability to locate suitable 
work suggest that locating any acceptable job might be quite difficult. Thus, without Job Service placements it is 
plausible that they would remain unemployed for substantial periods. 
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who most closely resemble those placed were unemployed 2.387 weeks less than average. This reduction 
is in keeping with expectations that those placed would find it easier to find jobs through use of the Job 
Service or through other means. 

Column 3 shows that use of the regression model reduces the measured effect of placements 
relative to being referred-but-not-placed by 4.744 weeks. This seems to us to be a large reduction that 
easily could be close to the amount of bias we would expect to have to take into account.^® Indeed, we 
regard the resulting estimates as conservative because they assume that the entire regression adjusted 
referral effect is due to selection bias, while it is plausible that at least half of the effect is due to the value 
of viewing listings and interviewing for jobs. 

A key reason for believing that referral effect are positive is that we doubt that job-seekers would 
waste their time using the Job Service over a considerable period if they felt that it was of no value. The 
data show that after the 10th week of unemployment 45 percent of claimants who are referred but not 
placed have actively used the Job Service in the 7 weeks prior to returning to work, and 20 percent 
actively used the Job Service in the preceding 12 weeks. This suggests that at least some of the measured 
effect represents more than a spurious correlation between use of the Job Service and otherwise 
unmeasured increased search intensity that is the true reason for those referred finding jobs more 
quickly. 



5.0 Estimates of Benefit-Cost Ratios 

While we recognize that we cannot demonstrate that our estimates of the per-person effects of 
referrals and placements are unbiased, we feel that these estimates are sufficiently indicative of the true 
effect to be used as the basis for comparing benefits to costs. Because of the uncertainty concerning the 
accuracy of these estimates we use highly conservative assumptions to estimate the cost-effectiveness Job 



Selection bias is a major problem in most studies of the effectiveness of employment and training programs 
because the per-person effects are small relative to plausible values of selection bias. By focusing on placement 
effects, which (we argue) should be quite large, the problem is considerably lessened. While we are unaware of a 
systematic attempt to measure the size of the bias, we believe that studies of job search assistance suggest the bias in 
no larger than our estimates of the positive referral effects. 

** An even greater persistence of use is found among placed job-seekers. Two-thirds of the claimants placed after 
the 10th week of unemployment had been actively using the Job Service to obtain referrals in the preceding 7 weeks, 
and about 20 percent had been actively using the Job Service in the preceding 1 2 weeks. We regard this persistence 
as additional evidence that the job-seekers who were placed recognized that the Job Service was providing a service 
of high value to them, and the measured effects should be large. 
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Service matching services. This evidence strongly suggests that services provided by the Job Service are 
highly cost-effective. 

Table 8 displays our benefit-cost estimates derived from use of the quasi-experimental estimates. 
Line 1 shows our estimates of the reduction in joblessness derive from being placed by Job Service 
referrals. . To translate the reduction in weeks into a dollar figure we multiple the reductions in joblessness 
by the average weekly earnings of the placed job-seekers subsequent to their returning to work. These 
calculations indicate that each placed job-seeker with a strong work record gained $1,872 in earnings and 
each placed job-seeker with a spotty work record gained $684. 



Table 8. Calculation of Benefit-Cost Ratio Using Quasi-Experimental Estimates for Washington 
Placement in 1998. 



Reduction in weeks of unemployment 
Earnings per week 
Increase in earnings 
Number placed in 1998 
Total increase in earnings 

Total earnings increase 
Cost of running the PLX 
Benefit cost ratio 



— work record 


— 


Strong 


Spotty 


7.2 


3.8 


$260 


$180 


$1,872 


$684 


11,144 


35,038 


$20,861,568 


$23,965,992 



$44,827,560 

$25,000,000 

1.8 



We then multiply the per-person earnings gains times the number of individuals placed in 1998 
based on the conservative assumption that all job-seekers with strong work records were claimants. 
Finally, we compare the total gains from all placements, which equals $44.8 million to the cost of 
operating the entire Washington State labor exchange system, which is $25 million. The resulting 
benefit-cost ratio is 1.8, which is quite respectful for any government sponsored emplo 3 mient and training 
program. 



We regard these estimates as conservative because the entire cost of running the Job Service is 
being compared to the value of placements alone. Yet as shown in table la, 60 percent of registrants 
receive some reportable services, while less than 10 percent of registrants are placed. Thus, even if the 
benefits derived from referrals not leading to placements, workshops and other forms of job search 
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assistance, various types of counseling and testing are very small, there total value would be quite high 
because they affect about 200,000 people. 

The key weaknesses in these estimates, however, is that we apply estimates derived from a small 
sample that may not be representative of the universe to all those placed, and the estimates for job-seekers 
with spotty records may be over-estimated. Nevertheless, benefits equal costs even if we assume the 
benefits to job-seekers with spotty work records are zero, as long as the omitted benefits are slightly 
greater than zero. 

Table 9 shows similar calculations using the per-person reductions in joblessness derived from 
the non-experimental estimates for UI claimants for 1995. The Oregon estimates are identical to the ones 
shown in table 3, but the Washington estimates are based on those that apply to 1994.*^ These figures are 
a bit lower than the average figures shown in table 3, but reflect economic conditions much more similar 
to those in 1995. 

We then multiplied the reduction in joblessness by the average weekly earnings in the year 
following the return to work to obtain estimates that on average each referral not leading to a placement 
increased earnings by $272 in Oregon and $416 in Washington, while each placement increased earnings 
by $1,136 in Oregon and $1,716 in Washington. 

Finally, we multiplied the per-person earnings gains by the number of claimants in the databases 
we used to produce the per-person estimates of the reduction in joblessness. This produced estimates that 
referrals plus placements increased earnings by $19.5 million and $24.3 million in Oregon and 
Washington, respectively. 

The Oregon figure equals about 80 percent of the cost of running its Job Service in 1995, and the 
Washington figure is about 10 percent greater than the cost of running its job service. However, because 
our estimates are limited to claimants, we feel that it is reasonable to compare the benefits to the cost of 
helping claimants alone. We made a rough estimate that 38 percent and 25 percent of the entire cost of 
running the Job Service in Oregon and Washington, respectively, goes to helping claimants. This 



We did not use Washington results for 1995, itself, because we felt that these estimates were not very accurate 
because we could not follow the return to work for a long enough period to properly exclude labor force dropouts 
from the comparison group. 
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Table 9. Calculation of Benefit-Cost Ratio Using Non-Experimental Estimators for Claimants in 
Washington and Oregon during 1995. 





Oregon 


Washington 


Reduction in weeks of unemployment 


per-referral 


1.1 


1.6 


per-placement 


4.6 


6.6 


Earnings per week 


$247 


$260 


Increase in earnings 


per-referral 


$272 


$416 


per-placement 


$1,136 


$1,716 


Number (according to administrative data) 


Referred in 1995 


34,704 


34,648 


Placed in 1995 


8,876 


7,533 


Total increase in earnings 


Due to referrals 


$9,429,225 


$14,413,574 


Due to placements 


$10,084,763 


$12,926,563 


Total earnings increase 


$19,513,988 


$27,340,137 


Cost of running the PLX 


$26,000,000 


$24,300,000 


Percent spent on claimants 


38.0% 


25.0% 


Benefit cost ratio 


Due to placements 


1.02 


1.26 


Due to referrals and placements 


1.98 


2.62 


Calculation of Percent of Funds Spent on Claimants 
Number (according to official statistics) 


Referrals in 1995 


143,697 


218,068 


Claimant referrals in 1995 


54,605 


54,517 


Claimant placements in 1995 


13,579 


11,524 


Claimant referrals as percent 


38.0% 


25.0% 



of all referrals 

Official referral as percent of 
administrative referrals 
Official placements as percent of 
administrative referrals 
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estimate is based on using official statistics to compare the ratio of all referrals to referrals made to 
claimants. 

Making these adjustment to cost lead to obtaining benefit-cost ratios of 1.02 and 1.26 for Oregon 
and Washington placements, respectively. The benefit-cost ratios increase to 1.98 and 2.62 in Oregon 
and Washington, respectively, when both referrals and placements are included. The Washington benefit- 
cost ratios are about 25 percent greater than those for Oregon both because benefits are greater and costs 
are less. 



However, the administrative data we used did not include all the referrals and placements 
reported for claimants in official statistics. We believe that much of the difference is due to our counting 
referrals and placements made only while claimants were collecting benefits, but the official statistics 
count as claimants those who exhausted benefits. It also appears that our Oregon database omitted 
substantial numbers of claimants, possibly because claimants who established a benefit year in 1995 and 
then collected benefits in that benefit-year in calendar-year 1996 were omitted from the file. 

If we reduce the cost to reflect the omission of services delivered to those claimants the benefit- 
cost ratios increase by over 50 percent in both states. Thus, we conclude that the benefits received by 
placed claimants alone are substantially greater than costs of providing those services, and would be 
greater still if all placements going to claimants were included in the datasets. Adding the benefits 
stemming from referrals not leading to placements increase the benefit-cost ratios substantially, and 
would do so even if selection and other measures reduced the true effect by large amounts. 



5.1 Additional Benefits and Costs 

The above benefit-cost calculations focus on estimation of the largest benefits — increased 
earnings, but these estimates do not provide estimates from the viewpoint of different members of society. 
In particular, they do not examine the loss of UI benefit payments that referred and placed claimants 
would have otherwise received or the possibility that the gains to these individuals come at the expense of 
a slower return to work by job-seekers who did not use the Job Service. We, therefore, examine 
reductions in UI payments and possible crowding-out effects in this section. 
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Table 10 displays the calculations needed to estimate reductions in UI payments. The first two 
lines show the estimated effect of referrals and placements on the number of UI payments. The next line 
shows the average payment, which is roughly half of the average weekly earnings of the claimants. We 
then multiply the per-person reduction in payments due to referral and placements times the average 
payment to estimate the per-person reduction in payments. Next we show the number of referrals and 
placements, which are identical to the figures in table 9. We then show the total reduction in payments by 
multiplying the incidence of referrals and placements times the per-event reductions. Finally, we 
compare the reductions in payments to the increase in earnings displayed in table 9, and the cost of 
providing services to claimants. 



Table 10. Calculation of Reductions in UI Payments- Using Non-Experimental Estimators for 1995 
in Washington and Oregon during. 





Oregon 


Washington 


Reduction in weeks of benefit payments 


per-referral 


0.00 


0.05 


per-placement 


2.30 


3.13 


Average weekly UI payment 


$124 


$130 


Decrease in payments 


per-referral 


so 


$6 


per-placement 


$283 


$407 


Number (according to administrative data) 


Referred in 1995 


34,704 


34,648 


Placed in 1995 


8,876 


7,533 


Total decrease in payments 


Due to referrals 


$0 


$326,012 


Due to placements 


$3,847,880 


$4,684,621 


Both 


$3,847,880 


$5,010,633 


Total decrease in payments as a 
Percent of the increase in earnings 


Due to referrals 


0% 


2.3% 


Due to placements 


38.2% 


36.2% 


Both 


19.7% 


18.3% 



Total decrease in payments as a 
Percent of costs of serving claimants 

Due to placements 38.9% 77.1% 

Due to referrals and placements 38.9% 82.5% 
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Overall, the reduction in UI payments equaled a bit less than 20 percent of the increase in 
earnings in both states. The reduction in UI payments is considerably less than the increase in earnings 
because: (a) payments equal only about one-half of earnings, and (b) payment reductions end at the point 
claimants exhaust benefits. Because many referrals and placements occur close to benefit exhaustion and 
those referred and placed claimants otherwise would be unemployed for long periods after exhaustion the 
per-person increase in earnings is more than twice as great as the per-person decrease in payments. 

It also is interesting that placements have a large effect on UI payments, equaling about 37 
percent of earnings reductions, but referrals have virtually no effect. This result provides further evidence 
that during the period benefits are paid claimants are quite reluctant to settle for any but highly suitable 
jobs. Thus, placements have a very strong effect presumably because they very well match the skills and 
interests of the claimants, but referrals not leading to placements appear to only have a strong effect as 
claimants have searched for long periods and begin to realize that they will have to settle for less desirable 
jobs than they hoped to find. 

Finally, it is noteworthy that the reduction in UI payments comes within about 80 percent of the 
cost of providing services to all claimants, even those who have exhausted benefits. The reduction in UI 
payments comes close to 40 percent of the cost of serving claimants in Oregon. The smaller percentage is 
largely due to our allotting a much larger fraction of total costs to serving claimants, and Oregon using its 
own funds to supplement its Wagner-Peyser grant. 

Another issue examined in our report was the extent to which the quicker return to work by 
claimants helped by Job Service referrals and placements slowed the return to work of job-seekers who 
did not receive this help. These crowding-out effects are potentially important because it could be the 
case that the benefits going to Job Service users largely come at the expense of other job-seekers. If this 
was the case the benefits in terms of reductions in UI payments would represent the major benefit to 
society, but the primary effect would simply be a redistribution of unemployment among job-seekers. 

The crowding-out analysis was performed by Professors Carl Davidson and Stephen Woodbury 
of Michigan State University (and in Dr. Woodbury’s case the Upjohn Institute). They used the 
Washington State estimates displayed in table 4 along with a variety of other data as parameters in a 
model they developed designed to measure the effect of various government interventions on total 

27 



O 

ERJC 



O 

c 



unemployment. They concluded that under a wide range of plausible assumptions the primary effect is a 
small decrease in unemployment (of about 0.08 percentage points), and a much smaller crowding-out 
effect (of about 0.061 weeks, which is an increase in duration of roughly 0.4 percent). 

That the reduction in unemployment is small is hardly surprising, given that the number of 
placements made by the Job Service represents a small fraction of all hires. However, the decrease is 
large enough to conclude that only 20 percent of the gains in earnings shown in table 9 come at the 
expense of other workers. 

Overall, we conclude that the primary beneficiaries of Job Service referrals and placements made 
to claimants accrue to the claimants themselves. The benefits are especially large per-person for those 
placed, even though about 40 percent of their gains in earnings comes at the expense of reductions in UI 
benefit payments. The benefits stemming from referrals are much smaller per-person, but large in total 
because about five times more claimants are referred than placed. Also, all most none of the earnings 
gains are offset by losses of UI benefit payments. 

Job-seekers who do not use the Job Service suffer very tiny per-person losses. However, because 
the number of job-seekers affected are large the total reduction in earnings equals about 20 percent of the 
gain experienced by placed and referred claimants. 

Society at large and employers also gain from reductions in UI benefit payments and the 
reduction in joblessness. UI benefit payment reductions alone come close to covering the costs of 
providing services to claimants in Washington, and cover a substantial proportion of the costs in Oregon. 
It is difficult to say how the benefits are distributed between employers and consumers because it is 
unclear how much of the cost of paying UI benefits is passed on to consumers. A reasonable guess is that 
the costs are split evenly between the two groups. Quantifying the benefits of reduced unemployment 
also is difficult, but even the benefits of the small reduction is likely to cover a large fraction of the costs 
of providing the benefits. 
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6.0 Summary 



The expert panel that reviewed our research agreed that the results summarized in table 1 1 
provide the best available estimates of the per-person reductions in joblessness and UI benefit payments 
resulting from Job Service referrals and placements. At the same time, they felt that the results are far 
from definitive because the quasi-experimental approach did not rule out the possibility that the estimates 
are biased because members of the quasi-control group were screened-out by employers and the sample 
may have been too small and unrepresentative of the universe. 

Thus, the panel concluded that the primary achievement of this work was to describe a procedure 
that could overcome the formidable estimation problems stemming from an inability to apply a random- 
assignment design. The panel felt that the best course would be to execute an improved survey, before 
the non-experimental results should be accepted as valid. 

The authors concur with the expert panel that it would be of enormous value to clear-up the issues 
they raised by conducting an improved survey. We are especially concerned about the validity of our 
results for job-seekers with spotty work records. However, we feel that the expert panel was taking an 
extreme position with respect to the results for claimants. Our analysis provided substantial evidence that 
the quasi-experimental results were not substantially biased by employer actions, and that any bias in the 
non-experimental results were small relative to the magnitude of the placement effects. 

In the absence of empirical information to the contrary, we therefore believe that our placement 
effect estimates that assume the positive referral effects solely are due to measurement error represent 
conservative estimates of the true effect. Based on this evidence it appears that the earning gains 
experienced by placed claimants more than cover the cost of providing services to this group. 

Also, because the costs of providing services are so small per-person, only a tiny fraction of our 
measure of the benefits of referrals not leading to placements need to represent the effect of these services 
rather than measurement bias. In addition, we assumed that no benefits accrue from any other service 
provided by the Job Service. This makes us believe that the total benefits are highly likely to 
substantially exceed costs. 
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Table 11. Summary of the Study Characteristics and Measures of PLX Benefits 







Back to work effect of: 






Data source 


Population 

studied 


Placement 
relative to 
referral 


Referral 
relative to 
no referral^ 


Total PLX 
benefits per 
year^ 


Benefit - cost 
comparisons^ 


Study-1 

Washington 
Mail Survey and 
Administrative 
Data for the first 
half of 1998 


A sample of 
587 individuals 
referred to PLX 
job openings 


7.2 weeks 
sooner for job 
seekers with 
strong work 
records 

3.8 weeks 
sooner for job 
seekers with 
weak work 
records 


Not 

examined 


$45 million for 
all 1998 PLX 
users from 
placements 
alone 


Annual cost 
$25 million 

Benefit-cost 
ratio 1.8 


Study-2 

Washington 
Administrative 
Data for 
1987-95 


A sample of 
328,815 spells of 
unemployment 
experienced by 
UI claimants 


7.7 weeks 
sooner 


2.1 weeks 
sooner 


$11 million for 
claimant 
placements 
alone 1987-95 

$25 million for 
claimant 
placements and 
referrals 
1987-95 


Annual cost $25 
million 

35 percent spent 
on claimants 

Benefit-cost 
ratio between 1 .2 
and 2.8 


Study-3 

Oregon 

Administrative 
Data for 
1995 


A sample of 
138,280 spells of 
unemployment 
experienced by 
UI claimants 


4.6 weeks 
sooner 


1.1 weeks 
sooner 


$15 million for 
1995 claimant 
placements 
alone 

$30 million for 
1995 claimant 
placements and 
referrals 


Annual cost $26 
million 

38 percent spent 
on claimants'* 

Benefit-cost 
ratio between 1.6 
and 3.1 



^ Referral effects measure the value of information obtained by viewing PLX listings and obtaining staff aid that 
improves the decision making of placed and non-placed PLX users. 

^ Study 1 uses published statistics to estimate the number of placements. Study 2 uses tabulations of person-level 
files to measure the number of placements and referrals. Study 3 uses both sources of information. Use of published 
data for 1995 raised benefit estimates for Study 2 to $42 million for placements and referrals together and $13 
million for placements alone. This increased the 1995 benefit-cost ratios to 4.5 for placements and referral and to 2.1 
for placements alone. 

^ Benefit-cost ratios are not adjusted for crowding-out effects. Their inclusion would reduce the ratios by about 20 
percent. 

^ Only 25 percent of Washington PLX costs went to referring claimants in 1 995. 



30 




31 



Acknowledgements 



This paper is based on research reported in Office of Workforce Security, Employment and 
Training Administration, U.S. Department of Labor, Occasional Paper 2000-06: Measuring the Effect of 
Public Labor Exchange fPLX) Referrals and Placements in Washington and Oregon . October 2000. 

An unusually large numbers of organizations and individuals contributed to this project and merit 
our thanks. First, and foremost we are grateful to the United States Employment Service Office of the 
Employment and Training Administration, U.S. Department of Labor for supporting this work through a 
grant to the Washington State Employment Security Department. 

Our primary contact within ETA was David Balducchi who provided invaluable aid and support. 
Equally important was the help we received from Jeff Jaksich in the “other” Washington. In addition, the 
Oregon Employment Department provided extraordinary support by processing its data to our 
specifications. Patrick Mclntire was our primary contact in Oregon. 

We are also grateful for the comments we received from our panel of technical experts: Burt 
Bamow, Johns Hopkins University; Arnold Katz, University of Pittsburgh; Bob LaLonde, University of 
Chicago; Jeff Smith, University of Western Ontario; Dan Sullivan, Federal Reserve Bank of Chicago; and 
Steve Woodbury, Michigan State University and the Upjohn Institute. We are also grateful to Dr. 
Woodbury and his colleague Carl Davidson at Michigan State University for executed the analysis of 
crowding out effects. 

Finally, we are grateful for the assistance we received from many of our colleagues at Westat for 
their help in executing the analysis and preparing the report for publication. 

Despite all of the help we received from others, any errors in the work are the responsibility of 
the principal authors, and the opinions expressed in this paper are solely those of the authors. 



31 




32 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 







EdticflllDnal Resouices InioiniQllon Ceniet 



I. DOCUMENT IDENTIFICATION: 







= ^ 

Author(s): ^ 2 A tSTTI^ 


Corporate Source: 


Publication Date: 


Tor. 


'ko^o 



II. REPRODUCTION RELEASE: 



In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, and 
electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if reproduction 
release is granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 



The sample sticker shown below will be The sample sticker shown below will be The sample sticker shown below will be 

affixed to all Level 1 documents affixed to all Level 2A documents affixed to all Level 28 documents 







PERMISSION TO REPRODUCE AND 






PERMISSION TO REPRODUCE AND 




DISSEMINATE THIS MATERIAL IN 




PERMISSION TO REPRODUCE AND 


DISSEMINATE THIS MATERIAL HAS 




MICROFICHE, AND IN ELECTRONIC MEDIA 




DISSEMINATE THIS MATERIAL IN 


BEEN GRANTED BY 




FOR ERIC COLLECTION SUBSCRIBERS ONLY. 




MICROFICHE ONLY HAS BEEN GRANTED BY 






HAS BEEN GRANTED BY 


























TO THE EDUCATIONAL RESOURCES 




TO THE EDUCATIONAL RESOURCES 




TO THE EDUCATIONAL RESOURCES 


INFORMATION CENTER (ERIC) 




INFORMATION CENTER (ERIC) 




INFORMATION CENTER (ERIC) 


1 




2A 




2B 



Level 1 Level 2A Level 2B 

Check here for Level 1 release, permitting reproduction Check here for Level 2A release, permitting reproduction Check here for Level 28 release, permitting reproduction 

and dissemination in microfiche or other ERIC archival and dissemination in microfiche and in electronic media for and dissemination in microfiche only 

media (e.g.. electronic) and paper copy. ERIC archival collection subscribers only 



Documents will be processed as irtdicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1 . 



Sign 

here, ^ 
please 



I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this 
document as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and 
its system contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other 
service agencies to satisfy information needs of educators in response to discrete inquiries. 



Signature; y 






Organization/Address:. 










Printed Name/PositiorVTitle:^^ / 

2-0 0} M n/ 






E-Mail Address; ^ 






FAX; 



Date: 



i; - . ^ 






(Over) 



III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 



Name: 



Address: 



V. WHERE TO SEND THIS FORM: 



Cend this fc.'m to the follov.'ing ERIC Clea.ringhcuse: Chery! GrOSSman 

Processing Coordinator 

ERIC Clearinghouse on Adult, Career, and Vocational Education 
Center on Education and Training for Employment 
1900 Kenny Road 

Columbus, OH 43210.1090 

However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 



